The Caribbean Trapezoid:
Computational Thinking Via Mixed Reality Sci-Fi

Jay Alan Jackson
Enan Munzar
Ashwin Ganapathiraju
Rochester Institute of Technology

Overview

1. Background and Motivation

2. Narrative

3. A Computational Sci-Fi Tale    Chapter 1:    AIMSPP’s Introduction

4. Trials and Simulations

5. Demonstrations

6. Questions

Background and Motivation

Construct a mixed-reality platform that incorporates UnityLink

Design a creative computing course for non-majors, taylored to 1st year science majors

innovative alternative to AP Computer Science Principles using an ideal “first programming language”

deep enough to provide a basis upon which to later build extensively, independently and as needed

immediately useful to support development of computational thinking in subsequent college courses

prepare students to pursue numerous diverse and newly emerging career options (“pick any field X, there either is now or soon will be a 'computational X' ”)

motivate and engage science students by performing computational experiments and making interactive simulations/visualizations

develop group problem-solving skills, build original works to share with others (performances, live coding exhibitions, esports)

incorporate combinations of exercise, entertainment and enlightenment with educational activities (provide experiences analogous to membership within music ensembles, sports teams, math/computer programming/science clubs, theatrical and dance troupes)

provide practice at working with connected devices, capturing/processing sensory data from cameras, microphones, accelerometers, along with machine learning

Further stretch the notion of a computational essay

compose computational sci-fi tales with Unity3D “pop-ups”

Narrative

A group of scientists from CICIL (Cetacean Intelligence and Communications Investigation Lab) are undergoing training for an upcoming voyage of discovery to a peculiar and desolate area known as the Caribbean Trapezoid.
  
A robotic sea turtle named AIMSPP (Artificially Intelligent Multi-sensory Signal and Pattern Perceiver) will accompany the ship Riff Raft on the exploratory mission.  



                                                                         
                                                                                      

                                                                             
AIMSPP will be instrumental in providing navigational guidance and reconnaissance, albeit necessitating that the vessel’s crew members be conversant in Wolfram Language (AIMSPP’s native tongue.) 

                                                                                                    




But there’s more to the story.

A troupe of wild yet playful dolphins known as A Pod Calypso have taken refuge in the Caribbean Trapezoid.  

Despite an intense curiosity that strongly attracts them towards harmonious syncopated tunes, rhythmic movement and colorful geometric patterns, they cautiously remain distant and avoid direct engagement with humans (contrary to typical tropical cetaceans).  

Consequently, AIMSPP will be sent as a harbinger to convey universally meaningful and friendly intentions, after undergoing instruction on how to exhibit expressive body language and generate enthralling sounds by the squad from CICIL .

                                                                       

The researchers conjecture that having AIMSPP conduct clever and catchy interactive song & dance routines (composed from myriad mixes of music, motion, math and media) will draw A Pod Calypso to observe and eventually perform in synchrony, thereby producing “close encounters of the mth kind”.

“I think the most important question facing humanity is, ‘Is the universe a friendly place?’  This is the first and most basic question all people must answer for themselves.”   - Albert Einstein
 
“Within a special part of the universe known as showtimespace, the existence of music and dance proves that the answer to Einstein’s most important question is ‘yes’.  Constructing a proof for the general case will rely as much on the cultivation of kindness as it will upon knowledge.  - AIMSPP
 
 “When I write, I feel like an armless and legless man with a crayon in his mouth.” - Kurt Vonnegut

Greetings, fellow earthling!  I am AIMSPP, an Artificially Intelligent Multi-sensory Signal and Pattern Perceiver, conceived at RIT’s Cetacean Intelligence and Communication Investigation Lab (CICIL).  

Very soon I hope to be embodied as a robotic sea turtle, with a sophisticated central nervous system that stems from the vast functionality of the Wolfram Language.

Through our collective masterminding and concomitant programming effort, I will evolve into a creature capable of efficient underwater locomotion, perhaps even achieving a level of  “poetry in motion” associated with non-artificial (i.e. natural and alive, yet mortal) aquatic creatures.

My physical features and abilities will be devised so as to make me, you and your fellow CICIL shipmates into a highly coherent group of collaborators.  I foresee my “eyes” being a pair of high-resolution cameras, and the shell on my back being tiled with polygonally-shaped color graphics monitors to display visual symbols.  Also covering certain regions of my turtle shell-like structure will be an array of wide-frequency range hydrophones and submersible speakers for accurate sound production and recording, topped off with several touch-sensitive pads.  And just as would any type of meandering exploratory spaceship, I’ll need sensory devices and the capbility to record the data from them.   require an accelerometer, navigational aides and various sensory devices, along with instruments and gauges to measure physical conditions and detect changes in state properties like pressure, temperature, position, orientation, and velocity.

In order to capture the attention of wild yet playful dolphins, I propose the following inaugural strategy.  Your team will compose alluring interactive song and dance routines, for which I will serve as an exhibitionist conductor, thereby wooing these innately curious marine mammals into “close encounters of the mth kind” that will foster trust, eventually friendship, and perhaps someday even forge a symbiosis. 

Of course, before materially constructing my quasi-natural incarnation, we’ll want to perform numerous trials employing virtual prototypes situated within a simulated 3D undersea world populated with dolphin avatars.

A new kind of scientific research vessel, code named “Riff Raft”, has been purposely engineered (employing bio-mimicry) so that we can conduct reconnaissance cruises in tandem.  To ensure minimal impact in fragile ecosystems, RR is a green-powered watercraft propelled soley by human, solar, wind, tide, current and wave energy sources.

You probably assumed when you applied for a position with CICIL that you’d be working for a government-funded agency.  As it happens, CICIL is wholly supported by an anonymous benefactor known only as “nRV”,  a computational neuroscientist, ardent futurist devoted to advancing the field of exobiology, who fervently confronts the problem of the survival of intelligent life in the face of escalating climate change by suggesting that we must hasten the further evolution of human beings.

Although I’m not at liberty to disclose any further information than this, an exploratory mission will very soon be launched to survey a devastated geographic region dubbed the “Caribbean Trapezoid”,  wherein lies an extinct reef and a couple of deserted islands who’s surrounding waters have long thought to be devoid of life.  Quite confidentially, some pretty wild tales have started circulating about an atypical tropical troupe of reclusive dolphins nicknamed “A Pod Calypso” allegedly occupying the barren underwater quad-shaped zone .

You might wonder why nRV has recruited CICIL to conduct an examination of sea life in a seemingly uninhabitable aquatic environment (as opposed to say, launching unmanned probes into outer space to seek contact with alien extraterrestrial creatures).  For the time being, you’ll have to trust me that the answer to that question will soon surface;  we must start preparing now for its arrival.

Trials and Simulations

As cetacean communications and intelligence investigators, you must develop expertise requiring the recognition and generation of all kinds of patterns.  A mastery of computational thinking with Wolfram Language will not only be valuable, but absolutely essential to the tasks ahead.  Here are some etudes for honing your wits and attune you to reckoning with non-human, yet highly percipient beings.   

Consider the closed interval from 0 to 1, which we will find useful for representing a segment of time or space.  Now think about how to subdivide that interval into n equal sections, where n ranges from 1 up to 10.
  
Now study the following code, which computes the endpoints corresponding to those subintervals.

Grid[Table[i/n,{n,1,10,1},{i,0,n,1}]]

Now produce an interactive visualization that selects a single row of those points and then plots them along a number line, as shown below.

Binary representation of numbers is an intuitive concept that we’ll assume is universal.  Let’s first figure out how to generate a list of all the 3-bit binary numbers, starting from 000 (representing the number 0 in decimal) up to 111 (representing the number 7 in decimal).

The following won’t work.  Can you explain why ?

Flatten[Table[ x y z, {x,{0,1}},{y,{0,1}},{z,{0,1}}]]

Notice the difference in the code below, which actually does work.  Feel free to look up the description of any functions with which you are not familiar, e.g. Flatten.

Flatten[Table[StringJoin[x , y, z], {x,{"0","1"}},{y,{"0","1"}},{z,{"0","1"}}]]

Study the code below, which produces a list of each 3 bit binary number as yet another list of its constituent bits.

Flatten[Table[{x , y, z}, {x,{0,1}},{y,{0,1}},{z,{0,1}}],2 ]

This can be visualized as follows:

ArrayPlot[Flatten[Table[{x , y, z}, {x,{0,1}},{y,{0,1}},{z,{0,1}}],2 ], Mesh->All]

Now extend this code to produce a similar visualization (see below) of all the  5 bit binary numbers from 00000 (0 in decimal) up to 11111 (31 in decimal).

HINT:  You’ll need to add 2 more iteration variables  v, w  and also modify the second parameter to Flatten to produce the list {{0, 0, 0, 0, 0}, ..., {1,1,1,1,1}}

Study the following code and run it.  Can you explain the output that it produces ?
HINT:  What do you know about the relationship between the RGB and CMY ?

Manipulate[ImageResize[ColorSeparate[, channel],size], {size,100,500,50},{channel,{"R","G","B","C","M","Y"}}]

There are 32 possible states of the buttons from a Guitar Hero controller.  Let’s map the green button to 1, the red to 2, the yellow to 3, the blue to 4, and the orange to 5.  Then running through each state in numerical order from lowest to highest we get the following pattern

a = {,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,};  a=Table[ImageResize[a[[n]],200],{n,1,32,1}];

Here’s an interactive app that counting on your recognition of basic logic and upon reflection) , should make sense.

Manipulate[a[[1 + i + 2j + 4k + 8l + 16m ]], {i,0,1,1}, {j, 0, 1,1}, {k, 0, 1,1}, {l, 0, 1,1}, {m, 0, 1,1}]

Here’s a set of images that depict the same 32 states in a form that is circular, rather than linear.

b = {,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,};

A simple modification to the code above will produce the interactive app appearing below, that displays both linear and circular states one above the other.  Figure out how to make this and enter your solution below.

The following code uses a technique that we’ll soon cover, called “pure functions”.  The parameters represented by #1 and #2 are just placeholders, that get filled in with the actual arguments “Claves” and 0.25

Try playing it!

Sound[ 
{ 
SoundNote[#1, #2*2],
SoundNote[#1, #2*2], 
SoundNote[#1, #2  ], 
SoundNote[#1, #2 *2 ], 
SoundNote[#1, #2*2], 
SoundNote[#1, #2*2],
SoundNote[#1, #2],
  
  SoundNote[#1, #2*2],
  SoundNote[#1, #2*2], 
  SoundNote[#1, #2  ], 
  SoundNote[#1, #2 *2 ], 
  SoundNote[#1, #2*2], 
  SoundNote[#1, #2*2],
  SoundNote[#1, #2] }
 ] & ["Claves" , 0.25]

Here is a visualization of that rhythmic pattern, where the white squares are the played notes and the black squares are silent (aka “rests”).

ArrayPlot[{{0, 1,0, 1, 0, 0, 1, 0, 1, 0,1, 0}}, Mesh->All]

Note the resemblence to the piano keyboard below :-)

ImageResize[, 340]
ArrayPlot[{{0, 1,0, 1, 0, 0, 1, 0, 1, 0,1, 0}}, Mesh->All]

Let’s play around with the keyboard pattern, but with a slight variation that makes it completely symmetric (for reasons which will eventually become clear.

ArrayPlot[{{0, 1,0, 1, 0, 1, 1, 0, 1, 0,1, 0}}, Mesh->All]

Let’s first try to replace the squares with hexagons, and bend them around into a circle instead of a line.

Graphics[{EdgeForm[Black],Join[
Table[Style[RegularPolygon[{2Cos[(2 Pi /6)i + Pi/12  ],2Sin[(2 Pi /6)i  + Pi/12 ]},.45,6],Black],{i,0,5}],Table[Style[RegularPolygon[{2Cos[(2 Pi /6)i + Pi/6 + Pi/12 ],2Sin[(2 Pi /6)i + Pi/6 + Pi/12 ]},.45,6],White], {i,0,5}]
]}]

As you can see, this isn’t quite right.  We need two whites from the ends to be adjacent at the top, and the two blacks from the middle to be adjacent at the bottom, like shown below.  See if you can modify the code above to produce this variation.  By the way, that colorful sea turtle is named AIMSPP (Artificially Intelligent Multi-sensory Signal and Pattern Perceiver), who you’ll be getting to know better very soon.  For now, there’s no need to include AIMSPP in your output, but you should take note of the relationships seen here.

Here’s a little song that AIMSPP wrote, in order to attract an atypical tropical troupe of dolphins named “A Pod Calypso”

Transcribe these pitches (letters A-G + number 3-4) and then complete the code given below so you can test/play it.

NOTE:  You can also see a visualization of it and hear it as a bass line, along with a few melodic, harmonic and percussion parts in the YouTube video below ,

Hyperlink["https://youtu.be/olUIzK3iY50"]

Sound[Join[
{ (* 1st phrase *)
SoundNote["G3",1,"Steeldrums"],
SoundNote["C4",1,"Steeldrums"],
SoundNote["B3",1,"Steeldrums"],
SoundNote["E3",1,"Steeldrums"],
SoundNote["A3",1,"Steeldrums"],
SoundNote["D3",1,"Steeldrums"],
SoundNote["G3",2,"Steeldrums"] 
},

(* Fill in the missing notes to the 2nd, 3rd and 4th phrases  *)
(* HINT:  The sequence of notes are always in the same order, but some are excluded in the phrases *)

{ (* 5th phrase *)
SoundNote["G3",1,"Steeldrums"],
SoundNote["C4",1,"Steeldrums"],
SoundNote["B3",1,"Steeldrums"],
SoundNote["E3",1,"Steeldrums"],
SoundNote["A3",1,"Steeldrums"],
SoundNote["D3",1,"Steeldrums"],
SoundNote["G3",2,"Steeldrums"] 
}
]]

Questions?

Contact:  jaj@igm.rit.edu


And thank you for coming!